Repository navigation
[trainer correctness] Backport Miles' fix for gradients kept between forward_backward calls - #257
Merged
Micah Yong (micahtyong) merged 9 commits intoNov 17, 2026
Conversation
Megatron ends every backward pass by reducing the gradient buffers. Miles keeps gradients until the optimizer step, so gradients from earlier forward_backward calls go through those reduces again. Two of them change the kept gradients: - Each buffer is multiplied by 1 / data-parallel size and summed over its data-parallel group. Dense buffers come back unchanged. Expert buffers are summed over the expert data-parallel group, so they come back multiplied by tensor_model_parallel_size / expert_model_parallel_size. - Weights replicated across tensor-parallel ranks, such as adapters on the MLA down-projections, are summed over those ranks, so they come back multiplied by tensor_model_parallel_size. Any forward_backward on the trainer triggers it, including another client's, because the reduces cover every adapter slot. Divide the kept gradients by those factors before each backward pass, so the reduces return them unchanged. A failed forward_backward marks the trainer unusable, so there is no partly scaled state to restore. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
keep_grads_through_reduce has to match every reduce that ends a backward pass. A reduce it misses would rescale kept gradients without any error. Each rank now reduces gradients of all ones once when the trainer starts, through the same call a backward pass makes, and fails if any gradient comes back changed. The error names the first weight that changed. The gradient accumulation tests now check one invariant per layout: kept gradients come back from the reduce unchanged. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
This was referenced Nov 11, 2026
Micah Yong (micahtyong)
marked this pull request as ready for review
November 16, 2026 18:22
Micah Yong (micahtyong)
requested a review
from Kailash Ranganathan (kailash109)
as a code owner
November 16, 2026 18:22
Merge main and replace the pre-divide and startup check with radixark/miles#4480's accumulate_gradients, which the miles-v0.2.0 image already runs: set kept gradients aside for each training pass so its reduce only sees the new gradients, then add them back. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rainer-expert-grad-accumulation # Conflicts: # CHANGELOG.md
# Conflicts: # CHANGELOG.md
# Conflicts: # src/spindle/backends/miles_runtime/actor.py
# Conflicts: # src/spindle/backends/miles_runtime/actor.py
Micah Yong (micahtyong)
deleted the
micah/nemotron-perf/trainer-expert-grad-accumulation
branch
November 17, 2026 02:40
This was referenced Nov 17, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A Tinker client can call
forward_backwardseveral times before oneoptim_step, and several clients' adapters can share one trainer. Miles keeps each adapter's gradients until itsoptim_step. Megatron ends every backward pass by combining gradients across GPUs over the whole gradient buffer, including the gradients kept from earlier calls. For most weights, combining them again changes nothing. For two kinds it does:tensor_model_parallel_size / (expert_model_parallel_size × expert_tensor_parallel_size).q_a_proj,kv_a_proj_with_mqa). Their gradients are summed over the tensor-parallel GPUs, so kept gradients come back multiplied bytensor_model_parallel_size.Each later
forward_backwardon a trainer therefore rescales the gradients waiting for anoptim_step, whether it comes from the same client or another one. Oneforward_backwardperoptim_stepwith no other client in between was always correct. Recipes affected: DeepSeek-V3.1, Kimi-K2.6, GLM-5.3 and GLM-5.3 Flash, gpt-oss-120b 128k, gpt-oss-20b 64k, Nemotron-3.5-Lightning 256k, Nemotron 3 Super and Ultra, Qwen3.5-397B and Qwen3.6-35B-A3B.Upstream Miles fixed this in radixark/miles#4480, and the
miles-v0.2.0image already has that fix. This PR backports the same function, unchanged, for the default Miles images: before each training pass it sets the kept gradients aside, so the pass's reduce only sees the new gradients, then adds them back. It is installed only for the default images and goes away when they move to Miles v0.2.0. It holds one extra copy of the gradient buffers during each training call, so it only runs on layouts where the reduce would rescale kept gradients: experts sharded differently from tensor parallelism, or sequence-parallel adapters with tensor parallelism above 1. Other layouts, and forward-only calls, are unchanged.This replaces #221, which fixes the same problem the same way.
E2E results
Setup: Nemotron-3.5-Lightning 30B weights on H200s, layout set by recipe overrides, LoRA rank 32, stock Tinker SDK. One client makes k identical
forward_backwardcalls and then oneoptim_step. The table shows the accumulated gradient divided by the single-call gradient, which should be k.Attention and shared-expert adapters were already right and stay at 2.00 and 4.00. With two clients, client A's routed-expert update after one call from client B was 1.99 times what it should be on TP2 / EP1; with this PR it is 1.00, also after two calls from B, after B's own
optim_step, and after forward-only calls.The extra copy costs memory and time. On TP2 / EP1 with four adapter slots it is 12.85 GB per GPU, and 128 rows of 1,600 tokens take 10.79 s in
forward_backwardinstead of 9.06 s (before, after). The stock TP2 / EP2 layout is not affected and skips the copy.